Improving Programming-by-Demonstration With Better Semantic Expression Thesis Proposal

The domain of applications that can be created with programming-by-demonstration (PBD) can be extended by improving the developers’ ability to communicate with the system. The techniques provided in this thesis will allow nonprogrammers to create a new variety of complete, interactive applications including many board games and educational software using PBD. A PBD software tool uses inferencing to induce programs by watching the developer demonstrate examples that show how the application should behave. Current systems reduce their scope or resort to having the developer program because they do not provide sufficient ways to express behaviors and the factors that affect them. Therefore, the goal of this thesis is to develop understandable forms of annotated expression and manipulation that help a system infer a broader range of behavior. To test these ideas, this proposal introduces a new system called Gamut that will present the techniques in a unified software tool. The first technique replaces the macro recorder method for demonstrating behavior used in other PBD systems with a technique called nudges. The developer demonstrates by correcting the system at important points during program execution and also using two nudge commands to communicate important situations. First, the Do Something! nudge causes the system to reconsider past learned behavior and try to generalize its knowledge to fit the current situation. Using the Stop That! nudge will point out improper behavior and generate negative examples. Second, Gamut will use a new deck-of-playing-cards metaphor to express concepts such as randomness, sequencing, and data storage. By constructing an appropriate deck, shuffling, sorting, and playing cards at key moments, developers can incorporate many effects not available without programming in other systems. Third, Gamut will improve communication about behaviors by making them more manipulable than in previous systems. Behaviors will be represented as small icons near the objects they affect. Using the familiar cut, copy, and paste commands, the developer can transfer behavior between objects. Determining how to make a behavior operate in the new context will be inferred automatically. An object’s state from the recent past will be represented as temporal ghosts in which objects become dimmed, translucent images. Many sorts of behavior refer to prior states such as a previous position or an old property value. The ghost objects will allow the developer to make explicit connections. Finally, to reduce the number of options the system must explore, the developer will be able to give hints by highlighting important objects and properties. A new inferencing algorithm will be created that will take advantage of the hints. By combining these techniques, Gamut will provide a rich medium for expressing developer intentions, fostering greater communication between the PBD system and the developer and enabling the developer to create highly interactive software with minimal programming expertise.